[CXX] Enable users to specify custom OrtSyncStream via RunOptions #26988

yuslepukhin · 2026-01-13T01:21:57Z

Description

Enable device stream override using RunOptions for a particular run.
The stream is restored after Run() completes.

Motivation and Context

Gpu InterOp requirements.
When enabled, the inference would run with the specified stream with proper synchronization with imported external synchronization facilities.

Copilot

Pull request overview

This PR enables users to specify a custom OrtSyncStream via RunOptions for GPU interoperability requirements. The stream override is applied for the duration of a single Run() call and automatically restored afterward, allowing proper synchronization with external GPU resources.

Changes:

Added RunOptionsSetSyncStream API to allow users to set a custom sync stream in RunOptions
Implemented stream override mechanism in DeviceStreamCollection that matches and replaces streams by device
Ensured automatic cleanup of stream overrides via DeviceStreamCollectionHolder destructor

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
include/onnxruntime/core/session/onnxruntime_c_api.h	Added `RunOptionsSetSyncStream` C API function declaration
include/onnxruntime/core/session/onnxruntime_cxx_api.h	Added `SetSyncStream` method to C++ RunOptions wrapper
include/onnxruntime/core/session/onnxruntime_cxx_inline.h	Implemented `SetSyncStream` inline method
include/onnxruntime/core/framework/run_options.h	Added `sync_stream` pointer field to OrtRunOptions
onnxruntime/core/session/ort_apis.h	Declared API implementation function
onnxruntime/core/session/onnxruntime_c_api.cc	Added new API to OrtApi struct
onnxruntime/core/framework/run_options.cc	Implemented `RunOptionsSetSyncStream` to set sync_stream field
onnxruntime/core/framework/device_stream_collection.h	Added `SetStreamOverride` and `ResetStreamOverride` methods
onnxruntime/core/framework/device_stream_collection.cc	Implemented stream override logic with device matching and automatic reset
onnxruntime/core/session/inference_session.cc	Integrated stream override before graph execution, with warning for parallel mode
onnxruntime/test/shared_lib/test_inference.cc	Added CUDA-based integration test for stream override
onnxruntime/test/framework/inference_session_test.cc	Added unit test for DeviceStreamCollection override behavior

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

include/onnxruntime/core/session/onnxruntime_cxx_api.h

onnxruntime/test/shared_lib/test_inference.cc

onnxruntime/core/session/inference_session.cc

include/onnxruntime/core/session/onnxruntime_c_api.h

onnxruntime/core/framework/device_stream_collection.cc

onnxruntime/core/session/inference_session.cc

onnxruntime/test/shared_lib/test_inference.cc

skottmckay

skottmckay

yuslepukhin · 2026-01-15T23:22:59Z

/azp run Linux QNN CI Pipeline,Windows GPU TensorRT CI Pipeline

azure-pipelines · 2026-01-15T23:23:08Z

Azure Pipelines successfully started running 1 pipeline(s).

yuslepukhin · 2026-01-16T19:00:53Z

/azp run Test Linux CUDA x64 Release,Test Linux TensorRT x64 Release,web_Debug / build_onnxruntime_web,web_Release / build_onnxruntime_web

azure-pipelines · 2026-01-16T19:00:59Z

No pipelines are associated with this pull request.

…crosoft#26988) ### Description Enable device stream override using RunOptions for a particular run. The stream is restored after Run() completes. ### Motivation and Context Gpu InterOp requirements. When enabled, the inference would run with the specified stream with proper synchronization with imported external synchronization facilities.

…6988) ### Description Enable device stream override using RunOptions for a particular run. The stream is restored after Run() completes. ### Motivation and Context Gpu InterOp requirements. When enabled, the inference would run with the specified stream with proper synchronization with imported external synchronization facilities. (cherry picked from commit 9f06181)

### Description This PR cherry-picks the following changes for the 1.24.0 release. ### Cherry-picked Commits | Commit | Commit Title | Author | |---|---|---| | 744e7fe | Add type definitions, registration, utilities for INT2/UINT2 support (#26824) | vraspar | | 530a1fb | [QNN EP] Add BFloat16 dtype support in QNN EP (#26987) | tirupath-qti | | 8e050d1 | Implement new experimental lookup-based matrix multiplication method(TMAC) (#26695) | vraspar | | 2d2ba6b | [MLAS/CPU EP] Improve performance of Silu activation path within the QuickGelu CPU kernel (#26753) | Hariharan Seshadri | | 1c02b79 | [QNN EP] Add support for handling 0-dimension for Concat Op (#27000) | Ashwath Shankarnarayan | | cc2b01b | Fix ClipQuantFusion crash when Clip has multiple input edges (#27016) | Edward Chen | | bbd3850 | [QNN EP] Support quantized BatchNorm with per-channel DQ params on QNN HTP (#26959) | qti-yuduo | | d8f0318 | Add API to get ep graph partitioning info (#26781) | Adrian Lizarraga | | b912b18 | [OVEP] OpenVINO EP Features and bug-fixes for ORT-1.24 - Follow up (#27007) | Preetha Veeramalai | | ba11af4 | [QNN-EP] Add MatMulNBits translation for GPU (#26340) | quic-tirupath | | c03c419 | [MLAS/NEON] Add dedicated kernel for depthwise convolution for ARM64 using NEON intrinsics (#26688) | Hariharan Seshadri | | e7dfd69 | [QNN-EP] Support alternate Layernorm fusion pattern in QNN preprocess (#26060) | qti-mattsinc | | 4013dc1 | Implement multithreading in qgemm_kleidi (#26301) | Melike Kaptan | | 9f06181 | [CXX] Enable users to specify custom OrtSyncStream via RunOptions (#26988) | Dmitri Smirnov | | cfccd64 | Added support for QMX kernels in MLAS (#26849) | qti-vaiskv | | 29d9b2f | Tweak external resource importer handle structs (#27040) | Scott McKay | | 9d108d0 | [QNN EP] Add QuickGELU operator support for QNN provider (#27034) | tirupath-qti | | b35688f | Add INT2 and UINT2 support for QDQ, transpose and cast ops (#27022) | vraspar | | 6d34aba | Introducing BF16 Pointwise NCHWc Convolution for Arm64 (#26838) | Rohanjames1997 | | 36017ad | [EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops (#27050) | Chi Lo | | 50a03e4 | Add a new pipeline for CUDA 13 nuget builds (#27023) | eserscor | | a0d4439 | [EP ABI] Update Graph_GetGraphView() implementation (#26711) | Chi Lo | | 34bb209 | [webgpu] Fix a bug for im2col (#27069) | Wenqin Yang | | 46e8d45 | [QNN EP] Add FusedMatMul operator support (#27044) | tirupath-qti | | 5e7e7a3 | Disable Float32_2Bits_Asymmetric_256x256 test (#27046) | vraspar | | 39f966e | Fix Doxygen documentation build error in onnxruntime_c_api.h (#27083) | Nick Eubanks | | 8a7a797 | Print tensor for new packed type of 2 bits (#27064) | Tianlei Wu | | 01f40e6 | Fix GPU JAR testing on Linux (#27011) | eserscor | | b6ed7f3 | Fix warning around ununsed code in QNN Android Emulator builds by clang (#27026) | Hariharan Seshadri | | d7daa45 | Raise the timeout for the ios simulator job (#27045) | Hariharan Seshadri | | 7e1d818 | upgrade emsdk to 4.0.23 (#27029) | Yulong Wang | | 347b990 | Fix failing mainline build on Arm64 linux (#27101) | Rohanjames1997 | | f481b17 | Add dedicated API to support extracting compatibility string from model metadata (#27015) | adrastogi | --------- Signed-off-by: Liqun Fu <[email protected]> Signed-off-by: bfilipek <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Jonathan Clohessy <[email protected]> Signed-off-by: Christian Bourjau <[email protected]> Signed-off-by: melkap01 <[email protected]> Co-authored-by: vraspar <[email protected]> Co-authored-by: tirupath-qti <[email protected]> Co-authored-by: Ashwath Shankarnarayan <[email protected]> Co-authored-by: Liqun Fu <[email protected]> Co-authored-by: carzh <[email protected]> Co-authored-by: Hector Li <[email protected]> Co-authored-by: carzh <[email protected]> Co-authored-by: Vrajang Parikh <[email protected]> Co-authored-by: Hariharan Seshadri <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Edward Chen <[email protected]> Co-authored-by: Yuduo Wu <[email protected]> Co-authored-by: Adrian Lizarraga <[email protected]> Co-authored-by: Preetha Veeramalai <[email protected]> Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: saurabh <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: Javier Martinez <[email protected]> Co-authored-by: Bartlomiej Filipek <[email protected]> Co-authored-by: bopeng1234 <[email protected]> Co-authored-by: Eric Crawford <[email protected]> Co-authored-by: MayureshV1 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Vishnudas Thaniel S <[email protected]> Co-authored-by: Yaru Du <[email protected]> Co-authored-by: Ryan Metcalfe <[email protected]> Co-authored-by: Dvoretckii, Mikhail <[email protected]> Co-authored-by: Pallavi Gupta <[email protected]> Co-authored-by: Jianhui Dai <[email protected]> Co-authored-by: Jiajia Qin <[email protected]> Co-authored-by: Changming Sun <[email protected]> Co-authored-by: Fei Chen <[email protected]> Co-authored-by: Yulong Wang <[email protected]> Co-authored-by: Akupadhye <[email protected]> Co-authored-by: Wang Ning <[email protected]> Co-authored-by: Maximilian Müller <[email protected]> Co-authored-by: Chi Lo <[email protected]> Co-authored-by: George Wu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Wanming Lin <[email protected]> Co-authored-by: quic-calvnguy <[email protected]> Co-authored-by: Jie Chen <[email protected]> Co-authored-by: xhcao <[email protected]> Co-authored-by: Wei-Sheng Chin <[email protected]> Co-authored-by: quic-hungjuiw <[email protected]> Co-authored-by: Ian Hunter <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kunal-vaishnavi <[email protected]> Co-authored-by: Jeff Kilpatrick <[email protected]> Co-authored-by: Jeff Kilpatrick <[email protected]> Co-authored-by: Scott McKay <[email protected]> Co-authored-by: Nenad Banfic <[email protected]> Co-authored-by: derdeljan-msft <[email protected]> Co-authored-by: n1harika <[email protected]> Co-authored-by: Ryan Metcalfe <[email protected]> Co-authored-by: Jaswanth Gannamaneni <[email protected]> Co-authored-by: Klimenko, Mikhail <[email protected]> Co-authored-by: liang <[email protected]> Co-authored-by: Garth Long <[email protected]> Co-authored-by: Jonathan Clohessy <[email protected]> Co-authored-by: Akshay Sonawane <[email protected]> Co-authored-by: Christopher Warrington <[email protected]> Co-authored-by: Ishwar Raut <[email protected]> Co-authored-by: Gaurav Garg <[email protected]> Co-authored-by: Xinpeng Dou <[email protected]> Co-authored-by: adrastogi <[email protected]> Co-authored-by: Aditya Rastogi <[email protected]> Co-authored-by: qti-hungjuiw <[email protected]> Co-authored-by: Pradeep Sakhamoori <[email protected]> Co-authored-by: Adam Pocock <[email protected]> Co-authored-by: mingyue <[email protected]> Co-authored-by: Susanta Bhattacharjee <[email protected]> Co-authored-by: Jozef Wludzik <[email protected]> Co-authored-by: Rajeev Sekar <[email protected]> Co-authored-by: Mayuresh M Varerkar <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Wenqin Yang <[email protected]> Co-authored-by: xieofxie <[email protected]> Co-authored-by: hualxie <[email protected]> Co-authored-by: Joshua Lochner <[email protected]> Co-authored-by: Christian Bourjau <[email protected]> Co-authored-by: Xiaofei Han <[email protected]> Co-authored-by: Dmitri Smirnov <[email protected]> Co-authored-by: chunghow-qti <[email protected]> Co-authored-by: Guenther Schmuelling <[email protected]> Co-authored-by: Jiawei Shao <[email protected]> Co-authored-by: czekun <[email protected]> Co-authored-by: Jaskaran Singh Nagi <[email protected]> Co-authored-by: quic-tirupath <[email protected]> Co-authored-by: qti-mattsinc <[email protected]> Co-authored-by: Melike Kaptan <[email protected]> Co-authored-by: Damien Dooley <[email protected]> Co-authored-by: qti-vaiskv <[email protected]> Co-authored-by: Rohanjames1997 <[email protected]> Co-authored-by: eserscor <[email protected]> Co-authored-by: eserscor <[email protected]> Co-authored-by: Nick Eubanks <[email protected]> Co-authored-by: adrastogi <[email protected]> Co-authored-by: Rohanjames1997 <[email protected]>

…crosoft#26988) ### Description Enable device stream override using RunOptions for a particular run. The stream is restored after Run() completes. ### Motivation and Context Gpu InterOp requirements. When enabled, the inference would run with the specified stream with proper synchronization with imported external synchronization facilities.

yuslepukhin added 3 commits January 9, 2026 15:57

Enable collection level override

9d29f87

Add C API

772348e

Add C++ API. Add API test.

afeba3f

yuslepukhin requested review from Copilot, nieubank and skottmckay January 13, 2026 01:21

Copilot started reviewing on behalf of yuslepukhin January 13, 2026 01:23 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

skottmckay reviewed Jan 13, 2026

View reviewed changes

yuslepukhin added 3 commits January 14, 2026 15:14

Address review comments, make sure override is used in a test

dc32102

Merge branch 'main' into yuslepukhin/run_options_stream

0bc0e43

Address bad merge

125829c

yuslepukhin marked this pull request as ready for review January 14, 2026 23:48

yuslepukhin added 3 commits January 14, 2026 15:49

Merge branch 'main' into yuslepukhin/run_options_stream

87233f3

Resolve conflict

b852285

Add platform to the lib name

9659a85

skottmckay reviewed Jan 15, 2026

View reviewed changes

onnxruntime/test/shared_lib/test_inference.cc Show resolved Hide resolved

Use V2 to append CUDA EP in the test

a69586c

skottmckay previously approved these changes Jan 15, 2026

View reviewed changes

yuslepukhin enabled auto-merge (squash) January 15, 2026 22:36

Resolve overload ambigiuty

4f10a71

yuslepukhin dismissed skottmckay’s stale review via 4f10a71 January 15, 2026 22:45

skottmckay previously approved these changes Jan 15, 2026

View reviewed changes

yuslepukhin added the release:1.24.0 label Jan 15, 2026

nieubank previously approved these changes Jan 16, 2026

View reviewed changes

yuslepukhin added 2 commits January 16, 2026 10:37

Merge branch 'main' into yuslepukhin/run_options_stream

2c8a96a

Fix merge problems

a45eff3

yuslepukhin dismissed nieubank’s stale review via a45eff3 January 16, 2026 18:44

yuslepukhin dismissed skottmckay’s stale review via a45eff3 January 16, 2026 18:44

nieubank approved these changes Jan 16, 2026

View reviewed changes

yuslepukhin merged commit 9f06181 into main Jan 17, 2026
90 checks passed

yuslepukhin deleted the yuslepukhin/run_options_stream branch January 17, 2026 02:52

skottmckay mentioned this pull request Jan 19, 2026

[TRT RTX EP] Add support for D3D12 external resource import #26948

Open

This was referenced Jan 21, 2026

1.24.0 release cherry-pick round 1 #27103

Closed

1.24.0 release cherry-pick round 1 #27104

Merged

tianleiwu removed the release:1.24.0 label Jan 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CXX] Enable users to specify custom OrtSyncStream via RunOptions #26988

[CXX] Enable users to specify custom OrtSyncStream via RunOptions #26988

Uh oh!

yuslepukhin commented Jan 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skottmckay left a comment

Uh oh!

skottmckay left a comment

Uh oh!

yuslepukhin commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

yuslepukhin commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[CXX] Enable users to specify custom OrtSyncStream via RunOptions #26988

[CXX] Enable users to specify custom OrtSyncStream via RunOptions #26988

Uh oh!

Conversation

yuslepukhin commented Jan 13, 2026

Description

Motivation and Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skottmckay left a comment

Choose a reason for hiding this comment

Uh oh!

skottmckay left a comment

Choose a reason for hiding this comment

Uh oh!

yuslepukhin commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

yuslepukhin commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants